"""
机器学习因子数据计算
先准备好文华指数数据：工业品指数，化工品指数
数据时间从2022年9月1日到2023年11月16日
1分钟周期
"""

import pandas as pd
import rqdatac as rq
from datetime import datetime, timedelta

whgy_weight = {
    "ru99.SHFE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.008542907,
            datetime(2022, 8, 19, 21, 0, 0): 0.008182723
         },
    "al99.SHFE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.004214709,
            datetime(2022, 8, 19, 21, 0, 0): 0.00570942,
         },
    "cu99.SHFE":
        {
            datetime(2018, 6, 1, 21, 0, 0):  0.004411733,
            datetime(2022, 8, 19, 21, 0, 0): 0.004807225,
        },
    "fu99.SHFE":
        {
            datetime(2018, 8, 17, 21, 0, 0):  0.06325973192,
            datetime(2022, 8, 19, 21, 0, 0): 0.053886656,
         },
    "TA99.CZCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.01216773938,
            datetime(2022, 8, 19, 21, 0, 0): 0.015919914,
        },
    "zn99.SHFE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.00639341319,
            datetime(2022, 8, 19, 21, 0, 0): 0.005471842,
        },
    "l99.DCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.01230114,
            datetime(2022, 8, 19, 21, 0, 0): 0.009547696,
        },
    "rb99.SHFE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.0400977762,
            datetime(2022, 8, 19, 21, 0, 0): 0.042272156,
        },
    "v99.DCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.018452457,
            datetime(2022, 8, 19, 21, 0, 0): 0.02114072,
        },
    "pb99.SHFE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.010468639,
            datetime(2022, 8, 19, 21, 0, 0): 0.011826181,
        },
    "j99.DCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.105055250,
            datetime(2022, 8, 19, 21, 0, 0): 0.118170888,
        },
    "ZC99.CZCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.31551223466,
            "exit": datetime(2022, 8, 19, 21, 0, 0)
        },
    "FG99.CZCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.1274655837,
            datetime(2022, 8, 19, 21, 0, 0): 0.126988744,
        },
    "jm99.DCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.1507113951,
            datetime(2022, 8, 19, 21, 0, 0): 0.141963338,
        },
    "bu99.SHFE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.0731544954,
            datetime(2022, 8, 19, 21, 0, 0): 0.062670772,
        },
    "i99.DCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.241032035,
            datetime(2022, 8, 19, 21, 0, 0): 0.247419635,
        },
    "pp99.DCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.01702231,
            datetime(2022, 8, 19, 21, 0, 0): 0.017148504,
        },
    "hc99.SHFE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.0542567618,
            datetime(2022, 8, 19, 21, 0, 0): 0.053333473,
        },
    "MA99.CZCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.0612434499,
            datetime(2022, 8, 19, 21, 0, 0): 0.065696018,
        },
    "SF99.CZCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.022856698298,
            datetime(2022, 8, 19, 21, 0, 0): 0.02455932,
        },
    "SM99.CZCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.023915528,
            datetime(2022, 8, 19, 21, 0, 0): 0.02264907,
        },
    "ni99.SHFE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.001551081783,
            datetime(2022, 8, 19, 21, 0, 0): 0.00140617,
        },
    "sc99.INE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.2899858612,
            datetime(2022, 8, 19, 21, 0, 0): 0.36488053,
        },
    "sp99.SHFE":
        {
            datetime(2019, 1, 4, 21, 0, 0): 0.03074927771,
            datetime(2022, 8, 19, 21, 0, 0): 0.030258476,
        },
    "eg99.DCE":
        {
            datetime(2019, 1, 11, 21, 0, 0): 0.024488445,
            datetime(2022, 8, 19, 21, 0, 0): 0.032237006,
        },
    "UR99.CZCE":
        {
            datetime(2019, 9, 9, 21, 0, 0): 0.080986060,
            datetime(2022, 8, 19, 21, 0, 0): 0.086157073,
        },
    "eb99.DCE":
        {
            datetime(2019, 11, 8, 21, 0, 0): 0.021580415,
            datetime(2022, 8, 19, 21, 0, 0): 0.019951759,
        },
    "nr99.INE":
        {
            datetime(2019, 12, 20, 21, 0, 0): 0.015910980,
            datetime(2022, 8, 19, 21, 0, 0): 0.013911752,
        },
    "SA99.CZCE":
        {
            datetime(2020, 1, 10, 21, 0, 0): 0.096960093,
            datetime(2022, 8, 19, 21, 0, 0): 0.092851855,
        },
    "sn99.SHFE":
        {
            datetime(2020, 3, 23, 21, 0, 0): 0.00127258139,
            datetime(2022, 8, 19, 21, 0, 0): 0.001108373,
        },
    "ss99.SHFE":
        {
            datetime(2020, 3, 23, 21, 0, 0): 0.00957286426,
            datetime(2022, 8, 19, 21, 0, 0): 0.01054253,
        },
    "pg99.DCE":
        {
            datetime(2020, 5, 15, 21, 0, 0): 0.0361575850,
            datetime(2022, 8, 19, 21, 0, 0): 0.041173494,
        },
    "lu99.INE":
        {
            datetime(2020, 7, 24, 21, 0, 0): 0.0541902512,
            datetime(2022, 8, 19, 21, 0, 0): 0.057094895,
        },
    "PF99.CZCE":
        {
            datetime(2020, 11, 6, 21, 0, 0): 0.027136559,
            datetime(2022, 8, 19, 21, 0, 0): 0.026200396,
        },
    "si99.GFEX":
        {
            datetime(2023, 3, 17, 21, 0, 0): 0.011263206,
        },
    "ao99.SHFE":
        {
            datetime(2023, 8, 18, 21, 0, 0): 0.064579891,
        },
    "br99.SHFE":
        {
            datetime(2023, 9, 22, 21, 0, 0): 0.014669679,
        },
    "lc99.GFEX":
        {
            datetime(2023, 10, 13, 21, 0, 0): 0.001016353,
        }
}


whhg_weight = {
    "ru99.SHFE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.010772714,
        },
    "TA99.CZCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.029804481,
        },
    "l99.DCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.017661862,
        },
    "v99.DCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.019300291,
        },
    "bu99.DCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.003850406,
        },
    "MA99.CZCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.063108922,
        },
    "pp99.DCE":
        {
            datetime(2018, 6, 1, 21, 0, 0): 0.017438787,
        },
    "eg99.DCE":
        {
            datetime(2019, 1, 11, 21, 0, 0): 0.030555754,
        },
    "UR99.CZCE":
        {
            datetime(2019, 9, 9, 21, 0, 0): 0.073842285,
        },
    "eb99.DCE":
        {
            datetime(2019, 11, 8, 21, 0, 0): 0.020110753,
        },
    "nr99.INE":
        {
            datetime(2019, 12, 20, 21, 0, 0): 0.014134168,
        },
    "SA99.CZCE":
        {
            datetime(2020, 1, 10, 21, 0, 0): 0.079437743,
        },
    "PF99.CZCE":
        {
            datetime(2020, 11, 6, 21, 0, 0): 0.020200093,
        },
    "br99.SHFE":
        {
            datetime(2023, 9, 22, 21, 0, 0): 0.01082205,
        }
}

whgy = {
    "start_time":{
        "si99.GFEX": datetime(2023, 3, 17, 21, 0, 0),
        "ao99.SHFE": datetime(2023, 8, 18, 21, 0, 0),
        "br99.SHFE": datetime(2023, 9, 22, 21, 0, 0),
        "lc99.GFEX": datetime(2023, 10, 13, 21, 0, 0)
    },
    "weight":{
        "ru99.SHFE": 0.008182723,
        "al99.SHFE": 0.00570942,
        "cu99.SHFE": 0.004807225,
        "fu99.SHFE": 0.053886656,
        "TA99.CZCE": 0.015919914,
        "zn99.SHFE": 0.005471842,
        "l99.DCE": 0.009547696,
        "rb99.SHFE": 0.042272156,
        "v99.DCE": 0.02114072,
        "pb99.SHFE": 0.011826181,
        "j99.DCE": 0.118170888,
        "FG99.CZCE": 0.126988744,
        "jm99.DCE": 0.141963338,
        "bu99.SHFE": 0.062670772,
        "i99.DCE": 0.247419635,
        "pp99.DCE": 0.017148504,
        "hc99.SHFE": 0.053333473,
        "MA99.CZCE": 0.065696018,
        "SF99.CZCE": 0.02455932,
        "SM99.CZCE": 0.02264907,
        "ni99.SHFE": 0.00140617,
        "sn99.SHFE": 0.001108373,
        "sc99.INE": 0.36488053,
        "sp99.SHFE": 0.030258476,
        "eg99.DCE": 0.032237006,
        "UR99.CZCE": 0.086157073,
        "nr99.INE": 0.013911752,
        "ss99.SHFE": 0.01054253,
        "eb99.DCE": 0.019951759,
        "SA99.CZCE": 0.092851855,
        "pg99.DCE": 0.041173494,
        "lu99.INE": 0.057094895,
        "PF99.CZCE": 0.026200396,
        "si99.GFEX": 0.011263206,
        "ao99.SHFE": 0.064579891,
        "br99.SHFE": 0.014669679,
        "lc99.GFEX": 0.001016353
    }
}

whhg = {
    "start_time": {
        "br99.SHFE": datetime(2023, 9, 22, 21, 0, 0),
    },
    "weight":{
        "ru99.SHFE": 0.010272788,
        "TA99.CZCE": 0.028911784,
        "l99.DCE": 0.016677319,
        "v99.DCE": 0.017566651,
        "pp99.DCE": 0.060959926,
        "MA99.CZCE": 0.01474617,
        "eg99.DCE": 0.028190849,
        "nr99.INE": 0.012451669,
        "UR99.CZCE": 0.067295446,
        "eb99.DCE": 0.018313188,
        "SA99.CZCE": 0.073249887,
        "PF99.CZCE": 0.020614505,
        "br99.SHFE": 0.010692271
    }

}

if not rq.initialized():
    rq.init("13570866213", "39314656")


def get_index_data(weight_dict, name, freq="1m"):
    """
    从20180601至20231116
    :param weight_dict:
    :param name:
    :return:
    """
    start_time = datetime(2018, 6, 1, 21, 0, 0)
    end_time = datetime(2023, 11, 16, 21, 0,0)
    rq_symbols = []
    rq_to_vt = {}
    #weight_dict = index_dict["weight"]
    #start_time_dict = index_dict["start_time"]
    for vt_symbol in weight_dict.keys():
        s, e = vt_symbol.split(".")
        rq_symbols.append(s.upper())
        rq_to_vt[s.upper()] = vt_symbol
    prices = rq.get_price(rq_symbols, fields="close", frequency=freq, start_date=start_time, end_date=end_time, expect_df=False)
    prices_fill = prices.fillna(method="ffill")
    if "m" in freq:
        minute_delta = int(freq.strip("m"))
        prices_fill.index -= timedelta(minutes=minute_delta)
    new_df = pd.DataFrame()
    for symbol in rq_symbols:
        vt_symbol = rq_to_vt[symbol]
        symbol_weight_dict = weight_dict[vt_symbol]
        exit_time = symbol_weight_dict.pop("exit", None)
        if not exit_time:
            exit_time = end_time
        symbol_data = None
        for t, weight in symbol_weight_dict.items():
            print(symbol, t, weight)
            if symbol_data is None:
                symbol_data = prices_fill.loc[t:exit_time, symbol] * weight
            else:
                symbol_data[t:] = prices_fill.loc[t:exit_time, symbol] * weight
        new_df[vt_symbol] = symbol_data
    new_df["wh_ind"] = new_df.mean(axis=1)
    prices.to_csv(f"D:\daily work\ml\wh_index2\\{name}_{freq}.csv")
    prices_fill.to_csv(f"D:\daily work\ml\wh_index2\\{name}_{freq}_filled.csv")
    new_df.to_csv(f"D:\daily work\ml\wh_index2\\{name}_{freq}_ind.csv")

def m_to_h():
    whhg_1m = pd.read_csv("D:\daily work\ml\wh_index2\\whhg_1m_ind.csv")
    whhg_1m["datetime"] = pd.to_datetime(whhg_1m["datetime"])
    whhg_1m["datetime"] = whhg_1m["datetime"].apply(lambda x: x.replace(minute=0))
    whhg_1m.drop_duplicates("datetime", keep="last", inplace=True)
    whhg_1m.to_csv("D:\daily work\ml\wh_index2\\whhg_1h_ind.csv", index=False)

def make_index_data(freq):
    gy1 = pd.read_csv(f'D:\daily work\ml\wh_index\\whgy_{freq}.csv')
    if freq=="1d":
        gy1.rename({"date": "datetime"}, axis=1, inplace=True)
    gy1["datetime"] = pd.to_datetime(gy1["datetime"])
    gy1.set_index("datetime", inplace=True)
    hg1 = pd.read_csv(f'D:\daily work\ml\wh_index\\whhg_{freq}_ind.csv')
    if freq=="1d":
        hg1.rename({"date": "datetime"}, axis=1, inplace=True)
    hg1["datetime"] = pd.to_datetime(hg1["datetime"])
    hg1.set_index("datetime", inplace=True)
    whgy1 = pd.DataFrame({"whgy": gy1["wh_ind"], "whhg": hg1["wh_ind"]})
    whgy1.to_csv(f"D:\daily work\ml\\wh_index_data_{freq}.csv")
